THREE-DIMENSIONAL IMAGE CAPTURING DEVICE 



BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a 

three-dimensional image-capturing device by which the 
three-dimensional shape and texture of a measurement 
subject are captured. 

2 . Description of the Related Art 

A three-dimensional image-capturing device, which 
detects the three-dimensional shape and texture of a 
measurement subject, is proposed. The three-dimensional 
image-capturing device sequentially captures a distance 
image, which represents the three-dimensional shape of the 
subject, and a texture image in a single image-capturing 
system; i . e . a unit of an optical system and an image sensor . 

However, in this type of three-dimensional 
image-capturing device, the image-capturing operations are 
cumbersome, since the distance image capturing operation 
and the texture image capturing operation are considerably 
different from each other. Further, it requires time for 
capturing a pair of distance and the texture images. 
Furthermore, since the distance image and the texture image 
are captured sequentially, it is impossible to obtain the 
distance image and the texture image simultaneously. 



SUMMARY OF THE INVENTION 

Therefore, an object of the present invention is to 
provide a three-dimensional image-capturing device that is 
able to obtain a pair of distance and texture images 
simultaneously through a biaxial or twin lens system, in 
which the optical systems are individually prepared for 
each purpose. Further, the object of the present invention 
is to provide a three-dimensional image-capturing device 
that can compensate for the distance image and the texture 
image . 

According to the present invention, a 
three-dimensional image-capturing device is provided that 
comprises first and second image sensors, first and second 
optical systems, a texture image capturing processor, a 
three-dimensional measuring processor, and a parallax 
compensation processor. 

The texture image capturing processor captures a 
still image of a subject in the first image sensor through 
the first optical system and the three-dimensional 
measuring processor measures distance information about 
the subject for each pixel of the second image sensor 
through the second optical system. The parallax 

compensation processor compensates for parallax between 
the first optical system and the second optical system by 
using the distance information. 
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Preferably, the device may further comprise a 
three-dimensional coordinate data generating processor. 
The three-dimensional coordinate data generating processor 
generates three-dimensional coordinate data by calculating 
5 three-dimensional coordinates for each point on the subject 
corresponding to each pixel of the second image sensor from 
the distance information. The parallax is compensated for 
by using the above three-dimensional coordinate data. 

In the first place, the three-dimensional 

i;3L0 coordinates may be calculated in a second coordinate system, 

W and then transformed to three-dimensional coordinates in 

a first coordinate system of which the origin is set at the 

ui focal point of the first optical system. In this case, the 

origin of the second coordinate system may be set at the 

i B 15 focal point of the second optical system. 

: a 
i xr 

C3 The parallax compensation processor may also 

execute a projection of each point on the subject 
corresponding to each pixel of the second image sensor onto 
the imaging surface of the first image sensor, so that a 

20 correspondence between pixels of the first and second image 
sensor is obtained. The projection is based on 

three-dimensional coordinates of each point on the subject, 
which are represented in the first coordinate system. 
BRIEF DESCRIPTION OF THE DRAWINGS 

25 The objects and advantages of the present invention 
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will be better understood from the following description, 
with reference to the accompanying drawings in which: 

Fig. 1 is a perspective view showing a camera 
provided with a three-dimensional image-capturing device 
of the present embodiment of the present invention; 

Fig. 2 is a block diagram showing an electrical 
construction of the camera shown in Fig. 1; 

Fig. 3 is a view showing the principle behind 
distance measurement ; 

Fig. 4 is a timing chart showing a distance 
measurement light beam, reflected light beam, a gate pulse, 
and the distribution of the accumulated amount of light 
received by a CCD; 

Fig. 5 show a flowchart of the process carried out 
in the camera ; and 

Fig. 6 schematically illustrates the arrangement of 
the first and second image-capturing systems. 
DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The present invention is described below with 
reference to the embodiments shown in the drawings. 

Fig. 1 is an external view of a camera-type 
three-dimensional device of a first embodiment of the 
present invention. With reference to Fig. 1, the 
three-dimensional device of the first embodiment is 
explained . 



On a front surface of a camera body 10, a 
photographing lens (first optical system) 11A, for 
capturing a texture image or a normal still video image of 
a subject, a photographing lens (second optical system) 11B, 
5 for detecting the distance information of the subject, and 
a viewfinder window 12 are provided. A good example of a 
texture image is a superfine color image . The distance 
information is detected for each of the image pixels so that 
the information can be regarded as a distance image, as will 
£3.0 be discussed later. In the middle of the upper surface of 
the camera body 10, a light-emitting device (light source) 

14, which radiates a laser beam, is mounted. A release 

in 

^ switch 15 and a liquid crystal display panel 16 are provided 

£3 on the left side of the light-emitting device 14 and a mode 

CO 

i s |5 change dial 17 is provided on the right side of the device 

□ 14. On a side surface of the camera body 10, a card slot 

i : 
:z as. 
I 

19 is formed, into which a recording medium, such as an IC 
memory card, is insertable. A video output terminal 20 and 
an interface connector 21 are also provided on the side 
20 surface. 

Fig. 2 is a block diagram showing an electrical 
construction of the camera of Fig. 1. 

An aperture 25A is provided in the photographing 
lens 11A. The opening degree of the aperture 25A is adjusted 
25 by an iris drive circuit 26A. A focusing operation and a 
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zoom operation of the photographing lens 11A are controlled 
by a lens drive circuit 27A. Similarly, an aperture 25B 
is provided in the photographing lens 11B and the opening 
degree of the aperture 25B is adjusted by an iris drive 
5 circuit 26B. Further, a focusing operation and a zoom 
operation of the photographing lens 11B are controlled by 
a lens drive circuit 27B. 

A CCD (first image sensor) 28A for capturing a color 
still image is disposed on the optical axis of the 
QO photographing lens 11A. Furthermore, on the optical axis 
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H of the photographing lens 11B, a CCD (second image sensor) 

;J 28B is disposed. The CCD 28B detects reflecting light of 

cn 

j'V a laser beam that is emitted from the light-emitting device 

i!3 and reflected by the subject. Subject images are formed 

f|5 on both the light-receiving surfaces of the CCDs 28A and 
□ 28B through the photographing lenses 11A and 11B, and an 

electric charge corresponding to the subject images is 
generated therein. An operation, such as an accumulating 
operation and a reading operation of the electric charge 
20 of the CCD 28A and 28B, is controlled by CCD drive pulse 
signals fed from a system control circuit 35 to the CCD drive 
circuits 30A and 30B. Electric charge signals, i.e. , image 
signals, read from the CCDs 28A and 28B are amplified by 
amplifiers 31A and 31B, and are converted from analog 
25 signals to digital signals by A/D converters 32A and 32B. 
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The digital image signals from A/D converters 32A and 32B 
are respectively subjected to a process, such as a gamma 
correction, in the image-signal process circuit 33, and is 
stored as digital image data in an image memory 34 . The 
5 iris drive circuits 26A and 26B, the lens drive circuits 
27A and 27B, the CCD drive circuits 30A and 30B, and the 
image-signal process circuit 33 are controlled by a system 
control circuit 35.. 

The digital image data are read from the image memory 
□ 0 34 and supplied to an LCD drive circuit 36, which is operated 

!. 1 

s ss- 

H in accordance with the digital image data, so that an image 

i s y 

1 \\ 

corresponding to the digital image data is displayed on an 
l± image indicating LCD panel 37. Further, an image processed 

1.3 

C3 in the system control circuit 35 can be displayed on the 

CO 

|1J5 image indicating LCD panel 37. 

ru 

£3 The digital image data read from the image memory 

34 is also transmitted to a TV signal encoder 38, so that 
the digital image data can be transmitted to a peripheral 
monitor device 39, provided externally to the camera body 

20 10, through the video output terminal 20. The system 
control circuit 35 is connected to an interface connector 
21 via an interface circuit 40. Therefore, the digital 
image data read from the image memory 34 can also be 
transmitted to a computer 41 connected to the interface 

25 connector 21 via an interface cable. Further, the system 
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control circuit 35 is connected to an image-recording 
device 43 through a recording medium control circuit 42. 
Therefore, the digital image data read from the image memory 
34 can be recorded in a recording medium M, such as an IC 
memory card, mounted in the image recording device 43. 

A light emitting element control circuit 44 is 
connected to the system control circuit 35. The 
light-emitting device 14 is provided with a light-emitting 
element, such as laser diode (LD) 14a, and an illumination 
lens 14b. The operation of the light emitting element 14a 
is controlled by the light emitting element control circuit 
44. The light emitting element 14a radiates a laser beam 
as a distance measuring light beam. The laser beam 
irradiates the entire measurement subject through the 
illumination lens 14b. The laser beam reflected by the 
measurement subject becomes incident on the photographing 
lens 11B. By detecting the reflected laser beam with the 
CCD 28B, information relating to the distance from the 
camera to the measurement subject is sensed at each pixel 
of the CCD 28 as an image signal. As will be discussed 
later , the distance from the camera 10 to the measuring 
subject that corresponds to each of the pixels in the CCD 
28B is derived for this image signal. 

A switch group 45, including the release switch 15 
and the mode change dial 17, and the liquid crystal display 
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panel (indicating device) 16 are connected to the system 
control circuit 35. 

With reference to Figs . 3 and 4 , the principle behind 
the distance measurement in the embodiment is described 
5 below. Note, in Fig. 4, the abscissa indicates time "t". 

A distance measuring light beam output by a distance 
measurement device B is reflected by the measurement 
subject S, and the reflected light beam is sensed by a CCD 
(not shown) . The distance measuring light beam is a pulse, 
ISO the width of which is "H". Accordingly, the reflected light 

a, 

w beam is a pulse, the width of which is XX H" , similar to the 

1:1 

distance measuring light beam. Therefore, a rise of the 
i' A pulse of the reflected light beam occurs after a rise of 

£3 the pulse of the distance measuring light beam by time 8-t 

CO 

3=5 (8 is a delay coefficient) . Since the distance-measuring 

ru 

Q light beam and the reflected light beam have both traveled 

a distance "r" between the distance measurement device B 
and the measured subject S, the distance "r" is represented 
as follows : 

20 r = 5tC/2 ... (1) 

wherein "C" is the speed of light. 

For example, by setting a condition in such a manner 
that the reflected light beam can only be sensed from the 
point where there is a rise of the pulse of the distance 

25 measuring light beam to a point prior to a fall of the pulse 
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of the reflected light beam, i.e. , by providing a gate pulse 
corresponding to a reflected light beam detecting period 
T, an amount "A" of received light from the reflected light 
beam becomes a function of the distance "r" . Namely, the 
greater the distance w r" (or the greater the time 5-t) , the 
less the received light amount A. 

By taking advantage of the principle described above, 
the three-dimensional measurement of the present 
embodiment is carried out in accordance with the received 
light amount A sensed in each of the photo-diodes which are 
two-dimensionally disposed on the CCD 28. Namely, from the 
received light amount A detected in each of the photodiodes 
or pixels , the distance from the camera body 10 to each point 
on the surface of the measurement subject S is sensed as 
an image signal (distance image) for each of the photodiodes 
or pixels. Thus, the distance data, which indicates the 
topography of the measurement subject S, is obtained from 
the image signal for each of the photodiodes or pixels. 

With reference to Fig. 5, a parallax compensation 
process for a gap between the texture image and the distance 
image of the subject is explained. The texture image is 
sensed in the first image-capturing system that comprises 
the photographing lens 11A and the CCD 28A. The distance 
image is sensed in the second image-capturing system that 
comprises the photographing lens 11B and the CCD 28B. Fig. 

10 



5 illustrates a flow chart of the entire process executed 
in the camera type three-dimensional image-capturing 
device of the present embodiment. 

The process represented by the flow chart of Fig. 
5 5 starts when the release switch 15 is fully depressed. 
Namely, when the release switch 15 is fully depressed, Step 
101 is executed so that the three-dimensional measurement, 
which is based on the principle of the distance measurement 
described with reference to Figs. 3 and 4, is carried out 
CBO by utilizing the light-emitting device 14 and CCD 28B of 
the second image-capturing system. By this, distance 

! : | 

**'f t information about the measurement subject, which 

j'V corresponds to each of the pixels in CCD 28B, is sensed as 

Q a distance image, and temporarily stored in the image memory 

CO 

j0)5 34 (distance image capturing process) . The capturing 

is : J- 

I y 

Q process for the texture image is then carried out in Step 

102. Namely, the texture image or normal still image of 
the subject is captured by the normal still video control 
of the CCD 28A, which is settled in the first image-capturing 

20 system. The captured texture image is temporarily stored 
in the image memory 34 (texture image capturing process) . 

In Step 103, the calibration for the distance image 
sensed in Step 101 is executed. Namely, distortion due to 
the second optical system or the photographing lens 11B is 

25 corrected. Similarly, in Step 104, the calibration for the 
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texture image sensed in Step 102 is executed, so that 
distortion due to the first optical system or the 
photographing lens 11A is corrected. 

In Step 105, the distance from the camera body 10 
to each point on the measurement subject, which corresponds 
to each pixel of the calibrated distance image, is 
calculated. Accordingly, the three-dimensional coordinate 
data of the measurement subject for each of the pixels is 
derived from the above distance. As it will be discussed 
later, the three-dimensional coordinate data, obtained in 
Step 105, may be expressed in the coordinate system for which 
the origin of the coordinate axes is set to the focal point 
O b of the second image-capturing system (refer Fig. 6) . In 
Step 106, the three-dimensional coordinate data of the 
measurement subject, which is obtained in Step 105, is 
transferred to the coordinate system for which the origin 
of the coordinate axes is set to the focal point O a of the 
first image-capturing system (refer Fig. 6) . Note that, 
the three-dimensional coordinate data corresponding to each 
of the pixels can be computed from the camera angle of view 
across the width and length of the second image-capturing 
system and the position of the pixel in the imaging surface 
of the CCD 28B. 

In Step 107, the correspondence of the pixels in the 
distance image to the pixels in the texture image is obtained 



with reference to the post- transferred three-dimensional 
coordinate data of the measurement subject. Namely, a 
position represented by three-dimensional coordinates that 
corresponds to each of the pixels in the distance image, 
the coordinates of which originate at the focal point 0 A/ 
is projected on the imaging surface of the CCD 28A, and then 
the corresponding pixel of the texture image is obtained. 
Accordingly, the correspondence between the pixels of the 
distance image and the texture image is achieved. The 
distance image is recorded in the recording medium M 
together with the corresponding texture image, and then the 
process is terminated. 

With reference to Fig. 6, the coordinates transfer 
process executed in the parallax compensation process in 
the present embodiment is explained. Fig. 6 schematically 
illustrates the arrangement of the first and second 
image-capturing systems . 

As shown in Fig. 6, in the present embodiment, the 
first and second image-capturing systems are arranged so 
that the respective optical axes L A and L B of the units 
intersect at a point P which is positioned at a predetermined 
distance from the camera. The distance image capturing and 
the texture image capturing are carried out on condition 
that the subject is arranged in the vicinity of the point 
P. The point P is on the line L 0 , which is a perpendicular 



bisector of the segment O a O b between the focal points O a and 
O b . Namely, the optical axes L A and L B intersect with the 
line L 0 at an angle 0. 

In Fig. 6, the focal point O a is the origin of the 
first coordinate system or the first Cartesian coordinates 
(X A/ Y A , Z A ) and the focal point O b is the origin of the second 
coordinate system or the second Cartesian coordinates (X B/ 
Y B/ Z B ) . The first Cartesian coordinates (X A/ Y A/ Z A ) are 
a right-handed coordinate system, wherein the coordinate 
axis X A is identical to the optical axis L A and the coordinate 
axis Z A is perpendicular to the surface of the figure in the 
upward direction. Similarly, the second Cartesian 

coordinates (X B , Y B , Z B ) are a right-handed coordinate system, 
wherein the coordinate axis X B is identical to the optical 
axis L B and the coordinate axis Z B is perpendicular to the 
surface of the figure in the upward direction. 

The distance data for each pixel of the CCD 2 8B, 
which is obtained by the distance measurement or 
three-dimensional measurement through the second optical 
system, is transformed to the three-dimensional coordinate 
data which is represented by the second Cartesian 
coordinates <X B , Y B , Z B ) for which the origin is at the focal 
point O b . For example, a point Q on the surface of the subject 
that corresponds to the point q B of the image capturing 
surface S B of the second image-capturing system, is 
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indicated by the three-dimensional coordinate data (x B , y B , 
z B ) of the second coordinates (X B/ Y B , Z B ) . 

The coordinate data (x B , y B/ z B ) can be transformed 
to the coordinate data of the first Cartesian coordinates 
(X A/ Y A , Z A ) by means of coordinates transformations, such 
as the 29 rotation of the second coordinates (X B/ Y B , Z B ) 
around the coordinate axis Z B/ toward the clockwise 
direction in the figure, and the succeeding translation of 
the focal point O b to the focal point O a . So that, when the 
coordinates obtained after the above rotation of the second 
coordinates (X B , Y B , Z B ) are denoted as (X A f , Y A f , Z A f ), and 
the three-dimensional coordinate data of the point Q in the 
coordinates (X A ? , Y A f , Z A T ) are denoted as (x A f , y A f , z A f ), 
the three-dimensional coordinates (x A/ y A/ z A ) of the point 
Q in the first three-dimensional coordinates (X A/ Y A , Z A ) 
are represented by the following equation; 



(2) 



wherein R is the rotation matrix that represents the above 
29 rotation around the coordinate axis Z B and (AX, AY, AZ) 
represents the vector from the focal point O b to the focal 
point O a . 

The correspondence between the point on the 

image-capturing surface S A of the first image-capturing 
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system and the point q B on the image-capturing surface S B 
of the second image-capturing system is obtained by 
determining the position of the point from the coordinate 
data (x A/ y A , Za) derived by equation (2) . As a result, the 
5 correspondence between the distance image and the texture 
image is obtained (Step 107) . 

Note that, in Fig. 6, although the optical axes L A 
and L B are illustrated as a straight line for convenience, 
the substantial optical axes L A and L B may be bent in the 
A3) first and second image-capturing optical system. 

The distortion of the first and second optical 
system, the arrangement of the first and second optical 
system, and the arrangement of the image-capturing surface 
and the focal point were previously fixed. Therefore the 
1J parameters required for the calibrations, the coordinate 

ru 

j; 3 transformations and the projection of the three-dimensional 

coordinate data onto the image-capturing surfaces were 
previously stored in memory. Namely, the above processes 
were carried out in accordance with these parameters. 

20 As described above, according to the present 

embodiment, the parallax in the distance image and the 
texture image, which are individually captured through the 
respective first and second optical systems, can be 
compensated . 

25 In the present embodiment, the optical axes of the 
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first and second optical systems are arranged so as to 
intersect with each other. The intersection of" the axes 
is not necessary, although the axes should be trained on 
the same subject. For example, the optical axes may be 
5 disposed parallel to each other. When the axes are parallel, 
only the translation operation is required for the above 
coordinates transformation and the rotation is not required. 
Namely, the transformation becomes easier. 

Although the embodiments of the present invention 

!<) have been described herein with reference to the 

w 

. ss ; accompanying drawings, obviously many modifications and 

! •! 

;i changes may be made by those skilled in this art without 

^ departing from the scope of the invention. 

Q The present disclosure relates to subject matter 

m 

UB contained in Japanese Patent Application No. 2000-254330 

m 

□ (filed on August 24, 2000) which is expressly incorporated 

herein, by reference, in its entirety. 
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